TITLE OF THE INVENTION 
ON-LINE IMAGE PROCESSING 
AND COMMUNICATION SYSTEM 

5 BACKGROUND OF THE INVENTION 

[0001] The present invention generally relates to miniPACS (Picture 

Archiving and Communications System) or teleradiology systems, specifically 
to miniPACS/teleradiology systems with remote volume data processing, 
visualization, and multi-user conferencing capability. In our previous patent 

10 application, United States Patent Application Serial No. 09/434,088, we 

presented a miniPACS/teleradiology system with remote volume data 
rendering and visualization capability. The present invention is directed to 
additional features and enhancements of the architecture described therein. 
[0002] Teleradiology is a means for electronically transmitting radiographic 

15 patient images and consultative text from one location to another. 

Teleradiology systems have been widely used by healthcare providers to 
expand the geographic and/or time coverage of their service and to efficiently 
utilize the time of healthcare professionals with specialty and subspecialty 
training and skills (e.g., radiologists). The result is improved healthcare 

20 service quality, decreased delivery time, and reduced costs. 

[0003] One drawback to some existing teleradiology systems, however, is the 
lack of the ability for radiologists to communicate interactively with their 
colleagues and referring physicians from disparate locations for the purpose of 
consultation, education, and collaborative studies. Collaboration is especially 

25 important for studies using volumetric images, where the ability to 

interactively manipulate the volumetric images and simultaneously view the 
processed images is essential for rapid and effective communications between 
multiple participants involved. 
[0004] There are numerous methods and systems providing multi-media 

30 network based conferencing capability. However, these methods and systems 
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only support shared viewing of texts, documents, and videos. Furthermore, a 
radiology conferencing system presents unique obstacles. For example, the 
size of data to be transmitted could be very large and the requirement on 
image (picture) quality could be very high. To be clinically useful, the 
5 transmission should be interactively "on-demand" in nature. There are on- 

going efforts to develop radiology conferencing capabilities for the 
communication of two-dimensional (2D) images. However, none of these 
systems supports interactive communication of volumetric/three-dimensional 
(3D) images. 

10 [0005] As a result, there exists a need for a miniPACS/teleradiology system 

with network based conferencing capability supporting synchronized 
distribution and viewing of interactively processed volumetric images. 
Further, there exists a need for an improved method and procedure for the 
management of multi-center trials involving volumetric images. 
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SUMMARY OF THE INVENTION 



[0006] The present invention provides a computer architecture for a 

client/server-based advanced image processing and rendering system. The 
present invention further provides a computer architecture to support multi- 

20 user concurrent usage of the processing server. The present invention includes 

a method and apparatus that combines the network-based conferencing 
capability with remote interactive advanced image processing capability. The 
present invention enables users from disparate locations to interactively 
manipulate images and simultaneously view the processed images in an 

25 independent or synchronized fashion. The present invention further enables a 

user to interactively view and manipulate the images without having to 
download the entire volumetric data set. The present invention also includes 
improved methods and procedures for radiology consultation and multi-center 
trial management involving volumetric images using the above-mentioned 

30 technology. 
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[0007] The present invention may be used for radiology consultation. In one 
step, the acquisition of 2D or 3D/volumetric image/data sets or retrieval of 
previously acquired image/data sets is performed. The volumetric data set 
could be three-dimensional in space, or two- or three-dimensional in space and 
5 one-dimensional in time, e.g., time-resolved spatial data sets. In another step, 

data is moved to a server, which could be the scanner workstation itself or a 
separate computer connected to a network, and which has the conferencing 
software. In another step, client software is initiated by a remote user/users. 
Each user is able to remotely access and manipulate the 2D as well as 

10 volumetric/3D images with full processing capabilities, including Multiplanar 

Reformat (MPR), Maximum Intensity Projection (MEP), Volume Rendering, 
Image segmentation, and etc. As described in the preferred embodiment, an 
user may send the image processing request, such as MPR request to the 
server, the server will render the images accordingly and send the result back. 

15 In another step, each user is able to interactively manipulate volumetric 

images without transferring the entire dataset, employing an "on-demand' 1 
image transmission method. 

BRIEF DESCRIPTION OF THE DRAWINGS 

20 

[0008] Fig. 1 depicts a block diagram of the present invention. 
[0009] Fig. 2 depicts an alternative diagram of the present invention. 
[0010] Fig. 3 depicts a flowchart of method of the present invention. 
[0011] Fig. 4 depicts a description of state parameters which may be used in 
25 one embodiment. 

[0012] Fig. 5 depicts a flowchart of the state parameter updating method. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
[0013] Fig. 1 depicts the teleradiology system described in our previous patent 
application, United States Patent Application Serial No. 09/434,088. The 
30 teleradiology system includes as data transmitting station 100, a receiving 
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station 300, and a network 200 connecting the transmitting station 100 and 
receiving station 300. The system may also include a data security system 34 
which extends into the transmitting station 100, receiving station 300, and 
network 200. Receiving station 300 comprises a data receiver 26, a send 
request 22, a user interface 32, a data decompressor 28, a display system 30, a 
central processing system 24, and, data security 34. Transmitting station 100 
comprises a data transmitter 16, a receive request 20, a data compressor 14, a 
volume data rendering generator 12, a central processing system 18, and, data 
security 34. 

[0014] Many image visualization and processing tasks (such as volume 
rendering) consist of multiple interactive sub-tasks. For example, visualizing 
a dataset consists of at least two steps (subtasks): 1) generating a processed 
image to be displayed; 2) displaying the image. In a client/server-based 
image processing system, some subtasks are performed by the client and the 
other by the server. Using the above example, generating the processed image 
to be displayed can be performed in entirety on the server, or, partially on the 
server and partially on the client. Displaying the processed image is 
performed on the client. 

[0015] Referring now to Fig. 2, a system is shown wherein, as contemplated 
in the present invention, several receiving stations 300a-e have access over a 
network 200 to a transmitting station. Each of the receiving stations 300a-e 
are structured similarly to the receiving station 300 shown in Fig. 1. The 
transmitting station may be considered the server and the receiving stations 
the clients to use the client/server terminology. 

[0016] Referring now to Fig. 3, a flow chart representing steps performed 
according to one preferred embodiment is shown. At step 401, one or more 
users initiate a session by logging in to the server from one of the receiving 
stations 300a-e. At step 402, one of the logged in users issues a command to 
form a conference, and identifies a list of users who may participate in the 
conference. The user who initiates the conference, e.g., the user at receiving 
station 300a shown in Fig. 2, may be designated as the conference "driver" by 



default. The conference driver may be, for example, a consulting radiologist. 
Other designated conference participants may join the conference. 
Alternatively, the driver may review the list of users logged in and select 
persons to participate in the conference. The other participants may be, for 
5 example, 3D technologists, other radiologists, referring physicians, or other 

healthcare personnel. The driver has the ability to accept or reject a request to 
join. Alternatively, the driver may designate that the conference is "open," 
Le. y that other users may join in without an express authorization being made 
by the driver. 

10 [0017] At step 403, the driver initiates a processing command from the client 

side. In a preferred operation, the driver, using interface 32, specifies: 1) at 
least one image data set to be visualized; 2) at least one data rendering method 
to be used; 3) the rendering parameters used by each rendering method, 4) 
data compression parameters, and 5) the data transmission parameters for 

15 controlling data transmission over network 200. Examples of state parameters 

are provided in Fig. 4. In particular, the driver may, via user interface 32, 
adjust rendering parameters, e.g., viewpoint, spatial region, and value range of 
the data to be rendered, and other settings. The techniques for setting and 
adjusting these parameters include 1) using preset protocols for some typical 

20 settings; 2) inputting a specific setting with a keyboard, a mouse and/or other 

input devices; and/or 3) interactive navigation using a mouse, a trackball, a 
joystick, a keyboard and/or other navigating devices. This driver may, via 
user interface 32, edit (including process) patient data, e.g., remove the bone 
structures, in a manner similar to the current volume data 

25 rendering/visualization systems. With the teleradiology system of the 

invention, driver can, via user interface 32, define and adjust data rendering 
methods and parameters, control what is to be rendered, transmitted and 
visualized next, and eventually obtain the final rendering result. A central 
processing system 24 on the driver's receiving station receives and validates 

30 the driver's request. The central processing system 24 then issues the request, 
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which is sent via send request 22 to transmitting station 100 through network 
200. 

[0018] At step 404, the central processing system 18 on the transmitting 

station 100 receives the request via receive request 20. Coordinated by central 
5 processing system 18, volume data rendering generator 12 accesses from 

image data source 10 the image data set which the user has specified, and then 
generates the data rendering result based on the data rendering method and 
parameters which the user has specified. The rendering result may be a 2D 
image, much smaller in size than the original data set. 

10 [0019] At step 405, the data transmitter 16 on transmitting station 100 

transmits the compressed data to data receiver 26 on receiving stations 300a-e 
which have sent a request for image data, f.e., on-demand, via network 200 
based on data transmission parameters which the user has specified. The on- 
demand feature of the present invention will be describe further in connection 

15 with Fig. 5. For the teleradiology system of the invention, the preferred 

transmission medium (i.e. 9 network 200) may be an intranet, the Internet 
(including the Internet2) or a direct dial-up using a telephone line with a 
modem. The preferred data transmission protocol is the standard TCP/IP, 
although the method may be adapted to accommodate other protocols. 

20 Furthermore, for some transmission media (e.g., the Internet2), user 400 can 

control certain aspects (e.g., the priority level, the speed) of data transmission 
by selecting transmission parameters via user interface 32. 
[0020] At step 406, the central processing systems 24 of the various receiving 
stations 300a-e coordinate the client-side processing. If needed, data 

25 decompressor 28 decompresses (or restores) the rendering result. The central 

processing system 24 may also perform further image processing and 
operations. The processing is performed in which the final image is computed 
based on the field of view and the image window/level (i.e., 
brightness/contrast) settings currently prescribed by the conference driver. 

30 [0021] At step 407, the display systems 30 at receiving stations 300a-e display 

the computed image and other parameters. Via user interface 32, the driver 
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may further modify parameters, including 1) the image data set to be 
visualized, 2) the data rendering method to be used, 3) the rendering 
parameters used, and 4) the data transmission parameters used. This process 
goes on until a satisfactory rendering and visualization result is obtained. 

[0022] The set of image processing and display parameters, collectively called 
state parameters, keep track of the effect of image processing, performed 
either at the server or at a client, and if needed, synchronize the display 
(viewing) of multiple users. Examples of state parameters are given in Fig. 4. 
Each time when a new subtask is performed, this set of the state parameters is 
updated at the server. Any further image processing and display task will be 
performed based on this set of updated state parameters. 

[0023] In one embodiment, the resulting images are "pulled" to the clients 
from the server. When a client with the driver authorization prescribes an 
operation and regardless of whether this operation is performed on the client, 
the server, or the both, the state parameters will be updated on both the server 
and the driving client to reflect the resultant change due to this operation. 
Other clients periodically compare their local copy of the state parameters 
with the copy on the server. If some differences are found that require 
updating the local display, that client will issue the update request. Again, 
depending on the division of subtasks, some requests are fulfilled by the client 
only, while the others require that the server sends updated image/information. 

[0024] Referring now to Fig. 5, the steps involved in state parameter updating 
will be described. State parameter updating is controlled by the client-side 
conferencing software running on receiving stations 300a-e. At step 501, a 
check is made with a system clock, or another timing source, to determine 
whether the amount of time that has elapsed since the last state parameter 
update, At, is equal to a predetermined timing parameter, P t , which determines 
the frequency with which the state parameters are updated. If At > P t , then 
step 502 is performed. If At < P t , then control returns to the beginning of the 
routine. For example, P t may be 0.25 to 0.5 seconds. At step 502, one of the 
receiving station 300 sends a request to the transmitting station for current 



state parameters associated with the current conferencing session. At step 
503, the receiving station 300 compares the state parameters which have been 
stored locally to the state parameters that are received from the transmitting 
station after the request made in step 502. If the client state parameters and 
the server state parameters are equal, then control returns to the beginning of 
the routine. If the two sets of parameters are not equal, this implies that 
additional subtasks have been specified by the conference driver, and the 
routine proceeds to step 504. At step 504, the client sends a request for new 
image data if the parameters that have changed indicate that new image data 
has been generated. On the other hand, if only state parameters relating to 
brightness or contrast level, for example, are changed, then no new image data 
need be requested, because this change can be processed on the data already 
stored at the client. At step 505, the client state parameters are set equal to the 
updated server state parameters. At step 506, At is set equal to zero. 
[0025] What has just been described is an on-demand image transmission 
method. Unlike the existing conference systems, image transmission occurs 
only when needed, and therefore, the network utilization efficiency is greatly 
improved. 

[0026] In an alternative embodiment, a "push" implementation is utilized. In 
the push implementation, state parameters are transmitted to the clients 
whenever they are changed. Also, new image data is transmitted if, as 
described above, the change in the state parameters required new server-side 
image processing. 

[0027] In another alternative embodiment, all remote conference participants 
may have already had the copy of the same data set on each of their local 
disk. This may be the case for training or educational applications in which a 
standard set of data is utilized. In this case, no image data transmission is 
required over the network. Based on the state parameters maintained on the 
server, the conferencing software running on each participant's computer will 
generate the new image using the local copy of the data and local computing 
resources and will synchronize the image display. This embodiment is useful 
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when the conference participants only have relatively-narrow bandwidth 
connection, such as a phone line, which is adequate to communicate the state 
parameters interactively, but not adequate for transmitting big data files, such 
as images, at rate allowing real time interaction. Updated state parameters in 
this embodiment may be transmitted to the clients either in a push 
implementation or a pull implementation. 
[0028] As part of the preferred embodiment, any participant in a conference 
may request to become the driver. Upon approval from the current driver, the 
driver privilege may be switched to the requesting participant. The new driver 
will then have the full control of the image/data set under study, Le., the 
ability to define new state parameters. The new driver, e.g., a surgeon, may 
fine tune the 3D model or other parameters to achieve the best view for his 
intended application. 
[0029] The present invention may also be applied to multi-center trial studies, 
when constant communication of comprehensive information including 
images and data are needed between multiple participants. One example is a 
Magnetic Resonance Angiography (MRA) multi-center trial. In an MRA 
study, a 3D volumetric data set, comprised of a stack of 2D images, is 
acquired. This 3D volumetric data set is processed to extract the vascular 
structure, while minimizing the interference of other unwanted structures. In 
order to select the highest quality protocols and design the most effective trial, 
the participants need to view not only the acquisition protocol and the original 
2D images, but also the processed 3D MRA images in detail. 
[0030] The existing multi-center trial procedures face several challenges. 
First, in order to reach consensus on trial protocols, principle investigators 
from participating institutions may need to travel to different locations 
multiple times, making this process time consuming and expensive. Second, 
the current procedure of site selection, training, and trial monitoring require 
frequent travel by the trial monitors to various participating sites, making this 
process heavily dependent on the trial monitors' travel schedule and 
availability. Third, the current process calls for transferring of all the patient 
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data/images to a centralized position, demanding significant amount of pre- 
work to modify studies headers and preserve patient privacy. 

[0031] The present invention provides an optimized method for multi-center 
trial management using the teleradiology conferencing technology. This 
method is designed to optimize the workflow and management of various 
tasks, such as protocol selection, training/education, trial monitoring, and data 
management for expert reading. 

[0032] The steps for future multi-center trial management using the present 
invention include: 

1) Using the teleradiology conferencing techniques described herein to 
choose a trial protocol; 

2) Subsequently using the training embodiment of the teleradiology 
conferencing techniques described herein to conduct interactive 
conferences hosted by the sites experienced in the selected protocols to 
provide training/education to other participating sites using the mechanism 
described in the above section; 

3) Using the teleradiology conferencing techniques described herein to 
conduct interactive conferences between the trial monitor and individual 
participating sites to review images, in order to assure quality and 
compliance during trial process; 

4) Using the teleradiology techniques described in our application, U.S. 
Serial No. 09/434,088, and the present invention to allow an expert reader 
to remotely review, and interactively process if needed, 2D/3D image sets 
store at centralized or disparate locations, without physically transmitting 
the entire image sets; and 

5) Reporting Expert reader will report blind read results using the 
integrated reporting tools provided by a system based on the present 
invention. 

[0033] While the present invention has been described in its preferred 

embodiments, it is understood that the words which have been used are words 
of description, rather than limitation, and that changes may be made without 
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departing from the true scope and spirit of the invention in its broader aspects. 
Thus, the scope of the present invention is defined by the claims that follow. 
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